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Variational Monte Carlo studies of a t-J model on an anisotropic triangular lattice 
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With the insulating phase of re-(BEDT-TTF)2X in mind, we study a t-J model on an 
anisotropic triangular lattice, where the hopping integral is t' in one of the three directions, 
using a variational Monte Carlo method. By changing the value of t' /t, we study the stability 
of superconducting (SC) states with d- and d+id-wecve symmetries and of an antiferromagnetic 
(AF) state. As t' /t decreases from 1, the stable state immediately switches from the d+id wave 
to the d wave. The AF state is stabilized from the normal spin liquid state for t' /t SS, 0.7 at 
half filling. We also take account of Nagaoka ferromagnetism and a phase separation. 
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1. Introduction 

Recent discoveries of superconductivity in k-(ET) 2 X 
(ET: BEDT-TTF) 1 and Na^CoOz • 2/H 2 2 are of high 
interest for some reasons. They are strongly correlated 
electron systems (2p-ir or 3c? orbitals) with quasi-two- 
dimensional lattice structures, like high-T c cuprates. 
However, they differ from the cuprates in that the lattice 
structure is essentially triangular, and thus the effect of 
frustration can play a leading role for superconductiv- 
ity as well as magnetism. For such lattices, we expect 
novel superconducting (SC) features, distinguished from 
the cuprates on the square lattice. In particular, identi- 
fication of pairing symmetry is a primary subject. 

Recently, we studied the stability of SC states for the 
t-J model on the isotropic triangular lattice, using a var- 
iational Monte Carlo (VMC) method, 3 which accurately 
treats the local constraint of no double occupation. Ac- 
cording to it, irrespective of the sign of t, a d x i_ y i+id xy 
wave is the most stable among various pairing symme- 
tries near half filling, in accordance with mean-field-type 
approximations. 4 At half filling, the simple d wave is de- 
generate with the d+id wave. In contrast, weak-coupling 
approaches which consider more realistic band structure 
of Na^-CoC^ • 2/H2O — a reduced single band 5 or multi 
bands 6 — led to different symmetries. 

In contrast, k-(ET) 2 X seems to be better described 
using single-band models on an anisotropic triangular 
lattice, where the hopping integral in two of the three 
directions is t and that in the remaining direction t' . Ac- 
cording to the band calculations for k-ET salts, 7,8 the 
value of t'/t is often near unity with t > 0. So far, the- 
oretical studies using the fluctuation exchange (FLEX) 
approximation for the Hubbard model 9 have concluded 
that the pairing symmetry is a simple d x i_ y i wave like 
the cuprates. The results of NMR 10 showed that the su- 
perconductivity is unconventional with nodes, but the 
pairing symmetry is not necessarily d wave. 

In this work, we extend our preceding study 3 to an 
anisotropic triangular lattice or a t-t'-J-J' model, which 
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connects the square lattice {t'/t = 0) and the isotropic 
triangular lattice (t'/t = 1). The main purpose of this pa- 
per is to compare the stability between the d- and d+id- 
wave pairing states, when the values of t'/t and carrier 
density S (1 — n) are changed. At half filling (5 = 0), 
the trial wave functions become insulating for the t-J 
model, but the gap parameter for the SC state re- 
mains finite; thus the favorable form of Ak can be de- 
termined for this singlet liquid (or RVB) state. Actually, 
k-ET salts, whose electron density is often at half fill- 
ing, bring about a superconductor-insulator transition, 
when their effective value of U/t (U: onsite repulsion) 
is increased through (chemical) pressure. 1 Since such a 
transition is properly described using a Hubbard-type 
model, as shown for the square lattice, 11 we will leave it 
for a coming publication. 

In addition, we also consider the possibility of Nagaoka 
ferromagnetism (FM) and a phase separation. For the 
isotropic case with t < 0, the Nagaoka FM defeats the 
SC states in a wide range of n and J/\t\ (0 < <5 ^ 0.96 
and < J/t ^ 0.6). 3 ' 12 It is still controversial whether 
the phase separation takes place or not for very small 
J/t and 8 ~ for the square lattice. We thus trace the 
development of these two states with changing t'/t. 

2. Formulation 

According to the comparison of experimental results 
with band calculations for the k-(ET) 2 X, 7 - 8 the ET 
molecules in the conducting planes are strongly dimer- 
ized. If each dimer makes a single lattice site, the con- 
ducting planes can be properly described with & t-J 
model on an anisotropic triangular lattice (or an ex- 
tended square lattice with next-nearest-neighbor pairs 
in only one diagonal direction [1,1]) as follows, 

H= U j P G (clc ja +H.c)p G 

+ Y J V ( S * ' S J ~ \ n i n o\ (!) 

(id) V J 
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Fig. 1. The most stable state among d- and d+id-wave and AF 
states is shown in the t' /t-8 space. Here, J/\t\ = 0.3, and the 
systems arc 10 X 10 and 12 X 12. (a) t > and (b) t < 0. 



where Pq = J^J^ (1 — n^n^). Here, Uj = t and = J 
for the nearest-neighbor-site pairs in two of the three 



lattice directions, and L 



t' and J,-. 



J' for the 



nearest-neighbor-site pairs in the remaining direction, 
and tij = J{j = otherwise. In view of the relation with 
the Hubbard model, J = At 2 /U, wc put J' /J = (t'/t) 2 . 
For k-(ET) 2 X, the value of t'/t is estimated as 0.5-1.1, 
and the electron density is mostly at half filling. In this 
paper, we consentrate on the case of n < 1, because a 
case of n > 1 can be mapped to a case of n < 1 with the 
different signs of t and t' through a particle-hole trans- 
formation. 

For this model, we perform VMC calculations, 13 which 
precisely treat the local constraint the t-J model im- 
poses. As a variational wave function, the Gutzwillcr 
type = Pg$) becomes a good starting point for t-J- 
type models. 14 We use ^ n = Pg&f for the normal state 
($p: Fermi sea), ^sc = -Pg'&bcs for the superconduct- 
ing state (^bcs 1 a BCS function with a k-dependent gap 
A k = Az k ), and * A f = Pg^af for the AF state ($af: a 
Hartree-Fock state with an AF gap A). In this paper, we 
focus on the d wave (z k = cosk x — cosk y ) and the d+id 
wave (z k = cosk x + e * 27r / 3 cos(k x + k y ) + e i47r / 3 cos k y ), 
because other symmetries are unstable in the regime of 
our concern for both the isotropic case 3 and the square 
lattice. 15 In ^sc and Waf, A is a sole variational param- 
eter, which becomes nonzero when has a long-range 
order, except for the case of ^sc at half filling (S = 0). 

In this study, we collect samples as many as 10 5 - 
10 7 , which reduce the error in energy to approximately 
10 _4 |i|. We use the systems of N s = 8 x 8-12 x 12 with 
periodic-antipcriodic boundary conditions and electron 
densities satisfying the closed shell condition. 

3. Results 

Let us start with the phase diagram in the t' /t-5 plain 
shown in Fig. 1, which is constructed by comparing the 
energies among the d- and d+id-w&ve and AF states for 
J/\t\ = 0.3. As studied in, 3 for the isotropic case (t'/t = 
1), the d+id wave is the most stable near half filling, but 
the d wave has an indistinguishably close energy for 5 = 
and low densities. When t'/t is reduced from 1, the stable 
symmetry rapidly switches to the d wave (solid circles), 
which prevails in a wide range of S and t'/t including the 
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Fig. 2. Energy reduction by the d- and d+id-w&ve spin liquid (or 
RVB) states and the AF state as a function of t'/t at half filling 
(<5 = 0). The systems are 10 X 10 and 12 X 12. Note that non- 
smooth behavior of E c chiefly stems from the discrete k-points of 
finite systems; when the value of t'/t changes, occupied k-points 
(or Fermi surface) discontinuously changes for the normal and 
AF states. 



square lattice, regardless of the sign of t. 

To consider the half-filled case more in detail, we plot 
the energy difference, E c — Esc — ^normal, as a function 
of t'/t in Fig. 2. Since both ^ n and ^sc are insulating for 
5 = 0, E c (< 0) indicates the energy reduction by forming 
a singlet liquid (or RVB) state. 16 When t'/t decreases, 
E c for the d wave considerably decreases, whereas E c 
for the d+id wave tends to increase. Thus, the d wave 
becomes widely stable, especially for t'/t ~ 0.5. On the 
other hand, for t'/t > 1, the d+id wave becomes more 
stable, inversely. The result of dominant d-w&ve symme- 
try is basically the same with those of the previous FLEX 
studies 9 for SC states. 

For the square lattice, the AF state ^>af is slightly 
more stable than the d-wave RVB state \l/sc at half fill- 
ing, and becomes unstable as soon as holes are doped. 15 
Even if t'/t is introduced as in eq. (1), the cases in which 
the AF state is the most stable are limited to half fill- 
ing with t'/t = and 0.1, as shown in Fig. 1. This is 
partly due to the fact that the t-J model tends to under- 
rate the stability of the AF state, because the AF state 
becomes stable in the weaker-coupling region than the d- 
wave does. 11 However, energy reduction by \I/af survives 
up to t'/t = 0.7 (see Fig. 2). Considering, in addition, 
that for t'/t = and < S ^ 0.1, a coexisting state 
of AF and d-wavc RVB orders is stable, 17 the d-wave 
RVB state at half filling may simultaneously exhibit an 
AF order in some range of t'/t(> 0). For t'/t > 0.7, it 
is probable that a Mott insulator with another spin or- 
der, e.g. a 120°-structure Neel order, 18 appears. We leave 
these issues for future studies. 

Next, we discuss Nagaoka FM, 19 which is a complete 
FM appearing in general for J/\t\ ~ and S ~ 0. For the 
square lattice, its region is fairly narrow, namely, < 
S <z 0.4 and J/t <^ 0.1. 20 In the isotropic case, the region 
of FM vanishes for t > 0, but considerably expands to 
< S £ 0.96 and J/\t\ £ 0.7 for t < 0. 3 ' 12 In Figs. 3, 
we depict phase diagrams for three values of t'/t. The 
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Fig. 3. Phase diagram in the 8-J/\t\ space for the phase sepa- 
ration and the complete FM for (a) t > and (b) t < 0. For 
t'/t = 0.3, the range of complete FM with t > is < 8 £ 0.2 
and J/\t\ ;C 0.015. The arrows on the vertical axis indicate the 
critical values of the phase separation estimated by the way (ii). 



boundaries are determined by comparing the energies of 
complete FM (spinless fermion) and the d wave (VMC), 
which is the most stable SC state. For t > 0, as t'/t 
increases, the range of FM rapidly reduces, and vanishes 
for t'/t — 0.5, whereas for t < 0, the range of FM rapidly 
expands at first, and becomes almost the same area as 
the isotropic case for t'/t = 0.8. Thus, FM does not seem 
to appear for t > 0, whereas for t < the Nagaoka FM is 
likely to expel the SC state from the realistic parameter 
region even for small t'/t. 

Finally, we consider a phase separation. Due to the ex- 
change term, the t- J- type models necessarily bring about 
a phase separation for large values of J/\t\. Here, we es- 
timate the boundary between the d-wave and phase sep- 
aration in two ways, (i) We suppose that the inhomogc- 
neous phase separates into the two hole densities, (5 = 
(half filling) and 6 = 1 (empty), and the energy of this 
state is approximated as _E PS = (1 — 5)E d , where the 
E d is optimized total energy of the d wave at half fill- 
ing. The boundary is determined by comparing E and 
E d (5). Although we use E d instead of _E cxact because the 
latter is not known for general values of t'/t, the results 
are connected without a marked discrepancy to the cases 
t'/t = 15 and l, 3 where £ cxact is available, (ii) We esti- 
mate the boundary near half filling, using the condition 
of intrinsic stability, d 2 E/dS 2 > 0, for the d wave. Near 
half filling, this approximation is better. 

The results of (i) and (ii) are summarized in Figure 3. 
For t > 0, the critical value J c /\t\ linearly increases as 6 
increases for large 5, and the slope is slightly suppressed, 
as t'/t decreases. For t < 0, as t'/t decreases, the slope of 
J c /\t\ becomes steeper. Regardless of the signs of t and 



the value of t'/t, the estimated value of J c /\t\ near half 
filling is small enough to be realistic. Thus, the possibility 
of phase separation remains near half filling, but this is 
not the case for intermediate and large values of 5. 

4. Conclusion 

We have studied a single-band t-J model on an 
anisotropic triangular lattice, based on variational Monte 
Carlo calculations. It is revealed that (1) within the su- 
perconducting states, the plain d x 2_ y 2 wave is dominant 
in almost the entire range of t'/t and 8 near half filling. 
(2) At half filling, the rf-wave RVB state is more stable 
than the d+id wave for t'/t < 1, while the relation is 
opposite for t' /t > 1; the AF state is stabilized from the 
normal spin liquid for t'/t ^ 0.7. (3) For t < 0, even when 
the value of t'/t is small, the Nagaoka ferromagnetism oc- 
cupies a wide range of S and J/\t\. (4) Regardless of the 
values of t'/t, the phase separation may arise near half 
filling, but not for intermediate and large values of 5. 

In this work, we have considered a t- J-type model for 
the insulating phase of k-(ET) 2 X. However, we need to 
tackle a Hubbard-type model to study a superconductor- 
insulator transition as well as a superconducting phase 
itself at half filling, like 11 for the square lattice. At the 
final stage in preparing this paper, we became aware of 
a VMC study for a Hubbard model. 21 
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